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Maya Angelou often said, 
“Once we know better, 
we need to do better” and 
Edgar Villanueva asks 
us to imagine, “What if?” 
(Decolonizing Wealth, 
2018). 


Effective implementation 
capacity is essential to 
improving education. 
The State Implementation 
& Scaling-up of Evidence- 
based Practices Center 
supports education systems 
in creating an implementation 
infrastructure for evidence- 
based practices benefiting 
students, especially those 
with disabilities. 
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The purpose of this brief is to provide examples from two state education agencies 

in the United States who took a different path to systemic change through the use of 

the Active Implementation Frameworks (Fixsen et al. 2013a; Metz & Bartley, 2012). 

Both states received intensive implementation-informed support from the 

State Implementation and Scaling-up of Evidence-based Practices (SISEP) Center. 

The aim of the SISEP Center is to establish implementation and scaling capacity in a 
State Education Agency (SEA), its regional entities, its districts, and their schools to 
continuously support teacher practice and improve outcomes. Under optimal conditions, 
a simultaneous process of horizontal and vertical alignment occurs: horizontal alignment 
across the SEA to reduce duplication of efforts at the state level that often translates 

into burdensome duplication of efforts in districts and schools; vertical alignment as the 
state facilitates and resources a co-creative process with its regional entities, districts, 
schools, and stakeholders to design an implementation infrastructure. Everyone’s efforts, 
horizontally and vertically, are focused on supporting state, regional, district, and school 
leaders, who in turn support continuous improvement of teacher skills to maximize 
academic and behavioral outcomes for all students. The benefits and challenges inherent 
in the separate approaches will be presented to offer the field a deeper understanding 

of organizational readiness for change and the importance of co-creation to engage 
benefactors from every level of the system to implement, sustain, and scale-up a 
practice with evidence (Metz, 2016; Tommeraas & Ogden, 2015). 


Using the mission-driven approach of the SISEP Center, organizations ask, “How?” 

five times (Blase, Fixsen, & Ryan Jackson, 2015); how will students benefit, how will 
teachers be supported, and how will school, district, and regional Implementation Teams 
be supported by the state? Across the globe governments and ministries, purveyors 
and practitioners, families and communities are asking, “What does it take, how do we 
effectively support use of a practice with evidence, and how do we do it?” Metz & Albers 
(2014) suggest it takes careful selection of a practice, use of stage-based activities, 

and the co-creation of an implementation infrastructure by teams with 

diverse perspectives and roles, so that data can be used for decision-making 

and continuous improvement across all levels of the system. Tommeraas & 

Ogden (2015) suggest it takes an active implementation approach 

(Fixsen et al. 2013a; Metz & Bartley, 2012), a stable infrastructure, and 

long-term governmental funding to sustain and take to scale effective use 

of a practice. Evidence from the field of implementation science suggests 

that systemic change of this magnitude requires careful attention to 

readiness and co-creation. 
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Developing readiness for change, or the developmental 
point at which an organization and its benefactors have 
the capacity and willingness to engage in system-wide 
change, can seem impassable at times (Fixsen et al. 
2013b). Inversely, it is invigorating when ‘dense webs of 
relationships’ engage in a co-creative process to build 
trust and understand the needs of all benefactors 

(Metz, 2016). Use of a co-creative process is predicated 
upon an explicit focus on assessing and understanding 
how various benefactors across a system build pathways 
for the use of evidence to improve outcomes for the 
concerned population. 


“This is where we find expertise and solutions, 
close to the ground, close to experience. 

The communities who have direct experience 
of an issue are by far the best experts on it.” 


—— Villanueva (2018) 


In the fall of 2014, with the intensive support of the SISEP 
Center, state #1 began to assess readiness and capacity 
(knowledge, skills, abilities) to co-create its implementation 
infrastructure (common training, coaching, and data use 
systems) to improve students’ mathematics outcomes 

and engage in deep systemic change. The focus was 
vertical alignment from the state to the region, district, 

and school to effectively implement state systemic change 
efforts and increase the percentage of students with 
disabilities performing at or above proficient in middle 
school mathematics. The use of the Active Implementation 
Frameworks was supported by one SEA office and two 
divisions to blend special and general education efforts 
and resources. After four years there is evidence to 
suggest that the vertical implementation infrastructure is 
durable, scalable, and sustainable. The structure, roles, 
and functions of teams from one state office to the region, 
district and school level continually support effective 
teacher practice and are improving mathematics outcomes 
for every child and student. Yet, four years later there is 
minimal readiness for simultaneous alignment of the SEA’s 
own offices, divisions, and major initiatives; organizational 
readiness is still lacking for systemic change to create 

a stable infrastructure and long-term governmental 
funding for sustainability of a common implementation 
infrastructure. 


Two State Transformation Specialists (STS) led the state’s 
implementation and scaling work in partnership with linked 
Implementation Teams comprised of state leadership, 
implementation workgroups, and implementation teams 

in two regions and two districts. This was the state’s first 
iteration of the Transformation Zone — a small slice of 

the state that will develop, use, study, improve, and then 
scale effective implementation efforts. Simultaneously, 

the state provided the resources for a team with diverse 
perspectives and roles to co-create a Usable Innovation 
for mathematics. The team began with the development 

of a Practice Profile, or the first three sections of a Usable 
Innovation (common philosophy, core components of the 
innovation, core components operationalized). 

This process ensured all educators and stakeholders 

could see themselves in the process and generate 
ownership for sustained use in districts. It defined a 
standard for mathematics practice in classrooms no matter 
what innovation (program, practice, etc.) a district selected, 
as long as it aligned with their Usable Innovation criteria for 
mathematics. Then, the state adopted the Observation Tool 
for Instructional Systems of Support (OTISS) developed 

by SISEP to measure high quality teacher practice at any 
grade level and within any content area (OTISS, 2014; 
Hattie, 2009). Adopting an evidence-based and 
research-validated fidelity measure completed the 

fourth and final section of a Usable Innovation. Teams 
collected and used fidelity data in aggregate to assess the 
implementation supports provided to teachers. The OTISS 
serves as an indicator of the extent to which the training, 
coaching, and other supports are sufficient, accessible, 
and in use by teachers. OTISS data is used to continuously 
improve these systems, rather than as a measure of 
evaluation. Once trust with teachers is developed, the data 
can be used with individual teachers to set professional 
goals. The protocol for collection and use of the fidelity 
data is developed with teachers who readily access the 
data (NIRN-SISEP Data System). 
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Once the Practice Profile was complete, some team 
members joined one of three workgroups responsible 

for developing the systems and measures for delivering 
training, coaching, and data use in schools. The goal was 
to clearly define what quality math instruction looked like, 
so it could be measured and fully supported in practice 
when the first schools and teachers were mutually selected 
for participation. Two Regional Education Agencies (REAs) 
and a large urban district mutually agreed to participate. 
The REA’s Regional Implementation Teams (RITs) were 
the link between the state and its districts. They developed 
the readiness and capacity of their districts, schools, 

and teachers to use the best available evidence from 
implementation and improvement science and research on 
mathematics best practice. When a barrier was identified, 
the barrier was lifted to the team with the authority and 
resources to solve that barrier. For example, RITs knew 
that using multiple forms of implementation data would 
require the development of a robust data dashboard. The 
state responded and developed a data dashboard. In the 
summer of 2018, the infrastructure developed in Learning 
Lab #1 (two REAs) was replicated and scaled-up in 
Learning Lab #2 (three REAs) who selected their districts 
and schools for initial implementation in the fall of 2019. 
Learning Lab #8 is currently in development. 


Teachers and school staff in the Transformation Zone make 
intentional use of data (implementation and outcome) 
monthly to report barriers to the team with the resources 
and authority to remove the barriers using Plan-Do-Study- 
Act Improvement Cycles (Deming, 1986). Central to this 
process is measuring capacity of the systems, activities, 
and resources required to initiate and sustain practitioner 
supports through action planning. State, regional, and 
district executive leadership commit to meeting every 
month to analyze capacity and implementation data 

in support of implementation teams and fidelity of 

teacher practice, in order to continuously improve the 
systems of support available to teams, school staff, and 
teachers. Figure 1 describes the capacity data from the 
state, one region, one of the region’s districts, and the 
district’s schools. The data suggests that the intentional 
development of a usable innovation and its implementation 
infrastructure (training, coaching, and data use systems) 
facilitated the district’s and its schools’ (1 elementary, 

1 middle) use of effective implementation supports 

for their chosen math program within the first year of 
engagement, exceeding the year 3 goal of 80% capacity. 
The state’s durable systemic change efforts from the state 
to the school are all in service to teachers who improve 
mathematics outcomes and close long-standing disparities. 


FIGURE 1 
State One Capacity data 2014-19 
(1 state, 1 region, 1 district, 2 schools, 15 teachers) 
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Figure 1 represents capacity data from one state, one 
region, one district, and their participating schools 

from 2014-2019. Baseline scores are typically low as 
Implementation Teams learn how to apply implementation 
and improvement science methods and tools. When 

the state’s total score dipped to 52% in 2016-17 due to 
administrative turnover, the regional and district total score 
was not affected. The trajectory of the region is much 
steeper as they develop the capacity of their districts. 
District growth is steady but slower as they prepare to 

use their chosen mathematics innovation in schools (a 
program, curriculum, etc. that aligns with the components 
of the state Math Practice Profile). The capacity of the 
teams at the regional and district level pay big dividends 
to schools. The middle and elementary school capacity 
assessment scores went from an average baseline score 
of 18% at the first administration in 2016-17 to 88% 

at the third administration, moving from exploration to 
initial implementation in one year. Note the district and 
elementary score dipped in 2018-19, as it should, as they 
prepared to scale-up with additional teachers and schools. 
Also, note the second dip in the state score. After the 

third change in state leadership and a state restructure in 
2018-19, the downward trajectory in state capacity did not 
affect the regional and middle school upward trajectory, 
suggesting the capacity of teams and the implementation 
infrastructure for mathematics may be durable and 
sustainable in the absence of the SEA horizontal alignment. 
Capacity to develop and sustain an implementation 
infrastructure horizontally across the SEA’s major programs 
and initiatives is reflected in the state’s total score; the total 
score never met the 80% three-year threshold due to low 
scores on items that measure SEA alignment. 


Page | 2 


Improved Student Math Outcomes 
Using a math screening measure, school staff monitored 
student improvement three times per year and observed 


improvement in the percent of students meeting proficiency. 


This was encouraging, however it typically takes three 

to five years to see improvement in state summative 
assessment data. Yet, this district showed improvement 
on the state summative assessment in the first year of 

the middle school’s implementation efforts. All students 

in this middle school increased Proficient performance on 
the state summative assessment from 2017 to 2018 by 
3.7% as well as two sub-group populations: Students with 
Disabilities (.3%) and African American (3.9%). A reduction 
in the performance for All Students at the lower measure 
of performance was noted as well as for four sub-group 
populations. Our hypothesis is that if capacity increases 
so should fidelity, and if fidelity increases so should student 
outcomes. 


Two State Transformation Specialists (STS) led state 
#2’s implementation and scaling work beginning in 2015, 
with the intensive support of the SISEP Center. The 

state initially focused on building horizontal alignment 
within the SEA to support the integration of federal 
identifications under two federal laws: the Individuals 

with Disabilities Education Act (IDEA, 2004) and Every 
Student Succeeds Amendment to the Elementary and 
Secondary Education Act (ESSA, 2015). Mirroring the 
federal structure, two separate teams within the SEA had 
traditionally ensured compliance with the requirements of 
each law. Under each law compliance-based reporting and 
monitoring requirements for districts and schools, including 
development of improvement plans, was operationalized 
by the two teams independently. The result of such an 
approach was a duplicative and burdensome compliance 
process for the states’ districts and schools. The State 
leveraged intensive coaching supports from SISEP to 
align the state’s continuous improvement process and 
requirements of ESSA and IDEA, grounded in the use 

of implementation science. The aim was to ensure that 
districts and schools would have more time to focus on 
developing capacity to support the use of evidence-based 
practices and improvement strategies to strengthen 
equitable outcomes for students, rather than navigating 
complex and duplicative compliance requirements. 


Given recent changes in federal legislation regarding 
ESSA and IDEA, the state recognized that large numbers 
of schools and districts would be identified as ‘needing 
improvement’ and thus would require support to ensure 
the selection and implementation of effective practices. 
Capacity building to install an infrastructure to align work 
within the agency began with their Title | and Special 
Education teams. In 2016, this horizontal alignment 
created an opportunity to explore the integration of 
continuous improvement processes with teams in other 
divisions, resulting in the expansion of cross-divisional 
teams. These teams train districts on root cause 

analysis, selection of evidence-based practices, educator 
effectiveness, response to intervention, coaching process, 
and collection of fidelity data. Stakeholders representative 
of various perspectives and roles in the state co-developed 
a continuous improvement process and a rubric informed 
by the Active Implementation Frameworks that ensured 
compliance with IDEA and ESSA, and guided districts in 
effective implementation of their improvement and student 
outcome goals. The state aligned their state-funded 
regional technical assistance teams from Title I, Special 
Education, Educator Development and Supports, and 
Response to Intervention center. This aligned approach to 
technical assistance required a braided funding system. 
So, four divisions came together to create an integrated 
funding contract to provide resources to their regional 

TA providers, who in turn would support districts and 
schools. Simultaneously, the state engaged in continuous 
improvement of their state data dashboard to create one 
consistent process for districts and schools to report and 
use their data. 
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Another system of support developed by coaching 
stakeholders in the state was a consistent process for 
delivering and measuring coaching supports, no matter 
what coaching system a district used (i.e., a Coaching 
Competency Practice Profile and measures). With these 
resources and supports in place, state-funded technical 
assistance providers attended targeted training (4 hours, 
one time per month) to receive training on use of the 
continuous improvement process, grounded in the use 
of selected Active Implementation activities. All regional 
entities in the state were represented and attended monthly 
targeted training to develop the skills and competency to 
support districts in the selection and implementation of 
effective practices to meet ESSA and IDEA goals. The 
state anticipates that this targeted training for all regional 
entities will develop readiness and capacity of REAs to 
mutually agree to participate in future iterations of the 
Transformation Zone. 


Beginning in 2017, or in year two, the state began 
exploring with regional entities to mutually agree to 
participate in a Transformation Zone and develop the 
capacity of two Regional Implementation Teams (RIT). The 
goal was development of a vertical linked teaming structure 
that leveraged existing regional technical assistance 
support systems funded by the SEA. The primary focus of 
the RIT was to provide intensive support to districts and 
schools to use the Active Implementation Frameworks 

to select and implement an effective practice to improve 
outcomes for all students and meet ESSA/IDEA goals. Two 
regions engaged in the work. Through the limited vertical 
slice of the system, the state began to use capacity data 

at the regional and state level to develop an enabling 
context. Unfortunately, mutual agreement to terminate the 
partnership in one region was reached in 2018 because 

of challenges associated with resource allocation and 
competing priorities within the organization. The SEA used 
the information learned to inform the capacity building in 
the remaining region and its two districts. 


The SEA used feedback from the remaining region in 
the Transformation Zone and capacity data to identify 
barriers and facilitators at the state and regional level. 
State leadership attended regional implementation team 
meetings to listen deeply and visibly demonstrate their 
commitment to systemic change in their districts and 
schools. When a barrier was identified, it was lifted up 
to the team with the authority and resources to solve the 


barrier — then, solutions were developed and put into 
action. A capacity report was developed monthly using 
multiple forms of data: qualitative feedback from regions 
and districts regarding barriers and facilitators, training 
effectiveness data on the understanding and use of the 
Active Implementation Frameworks, and training evaluation 
data. Data was used to strengthen capacity, competence, 
and confidence in the use of the Active Implementation 
Frameworks at the regional and district level. STSs 
delivered the capacity report to SEA cabinet members 
who were champions for the work every month. In turn, 
state updates were communicated to regions and their 
districts. The state and regional teams studied the Active 
Implementation Frameworks and began to put into use 
critical implementation practices in their Transformation 
Zone (e.g., mutual selection process for selecting districts, 
using the state Coaching Competency Practice Profile). 
Now the state is using lessons learned to scale the 
continuous improvement process to additional regions and 
districts in the Transformation Zone. 


Capacity Data for Action Planning 

Change at the SEA level takes time. “It is deviously difficult 
work” (Casciaro, Edmondson, & Jang, 2019). The SEA is 
learning that if simultaneous attention to the development 
of vertical regional-district-school structure is delayed, so 
too is valuable implementation data in the form of fidelity, 
training, coaching, and proximal student outcomes. In the 
initial two years of capacity building, the state capacity 
grew significantly, achieving the year 3 goal of 80% 
capacity at its fifth assessment administration. From that 
point forward, however, there were steady declines in the 
state-level capacity score. The successive capacity dips 
reflect the reduction in the number of regions contributing 
to the development of the regional structure, as well as 
fidelity to the use of the Active Implementation Frameworks 
to develop an implementation infrastructure for a usable 
innovation. Each district is responsible for the development 
of their Usable Innovation and training system for effective 
use of their selected EBP. 
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FIGURE 2 
State Two Capacity data 2015-2019 
(1 state, 1 region, 2 districts) 
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In 2018, the remaining RIT supported and coached the 
first districts in selection of a focus (e.g., literacy at the 
elementary level) and a specific program that would meet 
the identified need of students in the district. It became 
clear to the regions and districts that this was a very heavy 
lift and that it would take considerable time to select a 
program, ensure its usability, identify a evidence-based 
fidelity measure, and develop the associated training, 
coaching, and data systems for the district to effectively 
support their first schools. After months of deliberation, 
the region agreed upon use of effective teacher practice 
(Hattie, 2009) no matter what the area of focus, program, 
or grade level the district selected to study in their 
Transformation Zone. Now, the first districts could define 
and operationalize what they expected to see in any 
classroom and measure the practice for fidelity using 
their chosen fidelity measure. This decision resulted in an 
upward trend in the district capacity data at the second 
and third administration (2019), meaning the region 

is beginning to develop increased skills in their use of 
implementation practices at the district level to effectively 
support their schools and teachers. Due to the initial 
focus on horizontal alignment at the SEA, the state had 
limited district capacity data and no school capacity data. 
However, qualitative feedback from the region provided 
evidence of a barrier that had to be solved — regions and 


districts needed a common agreed-upon usable innovation. 


Outcomes 

Student outcomes are yet to be realized at the school level 
through intentional vertical alignment. We anticipate an 
upward trend in proximal outcomes at the end of the 
2019-2020 school year in the first district. In the meantime, 
the first school is using baseline fidelity data to examine 
and strengthen the systems of training and coaching 
available to teachers. Simultaneously, the RIT is supporting 
the first districts in the development of a Practice Profile 
that clearly defines and operationalizes effective teacher 
practice, and they are preparing for Exploration with an 
additional district. The SEA is also preparing for Exploration 
with additional REAs to establish the next iteration of the 
Transformation Zone that will benefit from the capacity 
developed in the current RIT, including systems, practices, 
and products aligned with the five Active Implementation 
Frameworks. 


Discussion 

The goal of the SISEP Center is to establish large-scale, 
sustainable, high-fidelity use of effective education 
practices in a state’s regions, districts, and schools 

while aligning and cohering the State Education Agency 
(SEA) offices and divisions. Research suggests that a 
simultaneous approach to systemic change from the lead 
organization to the practitioner and a decentralized strategy 
to effectively use a practice with evidence can reduce, 

if not completely eliminate, what is known as a scale-up 
penalty. Tommeraas & Ogden (2017) describe the scale- 
up penalty as a reduction of behavioral change (or end 
user outcomes) when attempting to scale-up a practice on 
a large-scale. They suggest organizations can eliminate 
the scale-up penalty if they use the Active Implementation 
Frameworks (Fixsen et al. 2013a; Metz and Bartley, 2012), 
establish a sustainable implementation infrastructure, 

and commit to long-term governmental funding by several 
ministries (or SEA offices in the US) who systematically 
evaluate practice (Ogden, Forgatch, Askeland, Patterson & 
Bullock, 2005). In this brief we presented two very different 
approaches to systemic change in two State Education 
Agencies supported by the SISEP Center. 
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State One established vertical alignment from the state to the school. Stakeholders co-created an implementation 
infrastructure so regions and their districts were ready to fully support teachers’ and coaches’ use of mathematics 
innovations in classrooms and produce evidence of improved mathematics outcomes. Yet, inattention to horizontal 
alignment creates fragmentation and duplication of SEA efforts. When multiple approaches to systemic change 

of a department’s major initiatives are presented without evidence of how they fit and complement one another, 
confusion manifests in regions and districts. The focus on implementation at the state remains in one Office. 
Horizontal alignment of the state department was not established in years one through four. This narrow focus 
creates siloed work and begs the asking of several questions: can use of a common language and way of work be 
scaled across the SEA to reduce duplication and fragmentation? Will a common way of work and language reduce 
confusion and undue burdens on regions and districts resulting from a layered history of best intentions? 


State Two developed significant capacity and commitment among state-level staff and leadership to align SEA 
major offices, divisions, and initiatives as they move away from previously siloed work to aligned data, teams, 
communication structures, and implementation processes. It has promoted an intentional alignment of the state’s 
strategic priorities and ways of work with a coordinated and aligned system of tiered supports for regions and 
districts using the Active Implementation Frameworks. There are also significant vulnerabilities. Namely, regional 
capacity development has been extremely slow due to a lack of focus on specific practices or innovations, where 
the onus of defining an effective innovation rests on the shoulders of regional, district, and school teams, thus 
requiring more time. State Two is four years into the active implementation journey with limited data to engage in 
practice-policy feedback loops to inform the SEA’s actions. This creates slow and limited scalability, and missed 
opportunities to co-create systems that are scalable with stakeholders. 


Questions remain for careful study and potential mid-course correction for both SEAs. First, will a 
focus on horizontal alignment facilitate the use of a common implementation approach at the SEA 
and also in its regions, districts, and schools? Will the presence of horizontal alignment and financial 
commitment across major SEA offices facilitate a greater rate of scaling with the creation of optimal 
conditions? Second, will a focus on vertical alignment create the conditions and lessons learned 

to sustain a stable vertical infrastructure with long-term governmental funding to take to scale a 
common implementation approach and effective use of any evidence-based practice? Will regions 
and districts have the capacity to develop and maintain an infrastructure for their identified effective 
instructional practices without SEA horizontal alignment? Third, will the SEA take the opportunity to 
learn and take action from the implementation efforts regionally and locally to make use of the AIFs 
as an approach for supporting strategic alignment both horizontally and vertically? Finally, fourth, 
will use of implementation and improvement science be de-scaled for another promising approach to 
systemic change? 
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Conclusion 


When considering readiness for systemic change, some organizations may find they do not have the capacity to 
transform the entire system simultaneously. The purpose of this paper was to provide two different examples of 
an SEA’s path to systemic change, and the facilitators and barriers to consider. Take a moment to reflect on three 
questions and consider your organization’s readiness for systemic change. 


»> First, if the approach taken in State One (vertical alignment) and State Two (horizontal alignment) 
were simultaneous, how would that change the collective capacity and efficacy of educators 
at all levels of the system? 


»> How would this affect the ability of an SEA to scale an innovation with a 0% scale-up penalty in a 
reasonable amount of time at the regional, district, and school level? 


»> How would this affect the ability of the SEA’s offices and major initiatives to leverage, align, and 
cohere, resulting in the reduction of duplication at the state to benefit regions, districts, and schools? 
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